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ABSTRACT 

Live  fire  training  keeps  warfighting  capabilities  at  peak  effectiveness.  However,  providing  realistic  targets  for  live 
fire  exercises  is  prohibitively  expensive.  The  United  States  Marine  Corps  uses  a  variety  of  target  proxies  in  live  fire 
exercises,  such  as  derelict  vehicles  or  piles  of  waste,  which  are  non-reactive  and  stay  in  fixed  locations.  Augmented 
Reality  (AR)  can  provide  realistic,  animated,  and  reactive  virtual  targets,  as  well  as  special  effects  such  as 
explosions,  for  real  world  training  exercises  with  no  significant  changes  to  the  current  training  procedure. 

As  part  of  USMC  Fire  Support  Team  (FiST)  training,  trainees  learn  how  to  call  for  fire  as  forward  observers  (FO). 
The  FO  determines  the  location  of  a  target  and  calls  for  fire.  After  the  round  is  fired,  an  instructor  determines  the 
effect  on  the  target,  and  the  FO  adjusts.  Initial  FiST  training  takes  place  on  a  scale  model  firing  range  using 
pneumatic  mortars,  which  is  where  we  inserted  an  AR  system. 

Our  system  provides  a  head-mounted  display  for  the  forward  observer  and  a  touch  screen  for  the  instructor,  each 
showing  virtual  targets  on  the  real  range.  The  observer  can  see  a  simulated  magnified  view  and  reticule  to 
determine  target  identity  and  location.  The  instructor  controls  the  targets  through  a  simple  interface.  The  FO  calls 
for  fire  and  a  real  round  is  fired.  The  instructor  sees  where  the  round  lands  in  the  augmented  touch  screen  view  and 
designates  the  effect  on  the  target.  The  forward  observer  sees  that  effect  and  adjusts. 

The  system  was  demonstrated  at  Marine  Corps  Base  Quantico  in  October  2004,  where  it  was  well  received  by 
mortar  trainees  and  instructors.  The  system  can  also  show  virtual  terrain  and  control  measures.  Future  plans  include 
testing  at  a  full-scale  live  fire  range  like  Twentynine  Palms  and  completing  a  Semi-Automated  Forces  (SAF) 
interface  for  more  intelligent  targets. 
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INTRODUCTION 

Live  fire  training  keeps  warfighting  capabilities  at  peak 
effectiveness.  However,  the  cost  of  procuring  real 
targets — only  to  be  destroyed — is  prohibitively 
expensive.  The  United  States  Marine  Corps  uses  a 
variety  of  target  proxies,  such  as  derelict  vehicles,  piles 
of  waste,  and  even  “pop  up  targets,”  all  of  which  are 
non-reactive,  stay  in  fixed  locations  from  year-to-year, 
and  often  do  not  resemble  the  real  targets.  Trainees 
simply  do  not  get  the  opportunity  to  fire  live  rounds  at 
realistic-looking  and  moving  targets.  However, 
Augmented  Reality  (AR)  can  help  by  merging  virtual 
entities  with  the  real  world  for  training  exercises.  In 
this  article,  we  describe  an  AR  system  that  provides 
virtual  targets  for  training  of  USMC  Fire  Support 
Teams. 

Augmented  Reality 

In  an  AR  system,  the  user  wears  a  tracked  see-through 
head-mounted  display  with  stereo  headphones  that  is 
connected  to  a  computer  containing  a  database  of 
spatial  information  related  to  the  venue  of  the  training 
exercise.  By  measuring  the  user’s  position  and  view 
direction  in  the  real  world,  three  dimensional  computer 
graphics  and  spatially  located  sounds  are  displayed  to 
appear  to  exist  in  the  real  world.  A  miniaturized  and 
ruggedized  computer,  batteries,  and  wireless 
networking  make  the  AR  system  man  portable  (Julier 
et  al  2000).  Figure  1  shows  a  mobile  AR  prototype 
system.  In  the  case  of  AR  for  training,  the  virtual 
information  overlay  consists  of  realistic  three- 
dimensional  renderings  of  entities:  individual 
combatants,  tanks,  planes,  ships,  and  so  on. 

Entities  in  Training  Simulations 

Entities  in  training  exercises  fall  into  one  of  three 
categories:  live,  virtual,  and  constructive  (USDoD 


1995).  Live  entities  are  real  people  and  vehicles 
participating  in  a  training  exercise;  virtual  entities  are 
human-controlled  players  in  virtual  worlds;  and 
constructive  entities  are  driven  by  algorithms  in 
computer  simulations.  AR  provides  a  natural  way  for 
all  three  types  to  mix  together.  Live  entities  observe 
virtual  and  constructive  entities  through  the  AR 
system.  Interactions  such  as  the  user’ s  movements  and 
weapon  usage  are  conveyed  from  the  AR  system  back 
to  the  constructive  and  virtual  simulation  systems.  Fire 
Support  Team  Training  is  a  prime  venue  to  insert 
virtual  and  constructive  entities  to  combine  with  live 
fires. 


Figure  1.  A  Wearable  Augmented  Reality  System 
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Fire  Support  Team  (FiST)  Training 

The  USMC’s  Fire  Support  Team  training  begins  with 
small-scale  (1:40)  pneumatic  mortars  on  a  50m  x  75m 
field  at  Quantico,  simulating  a  2km  x  3km  area  of 
operation.  The  purpose  of  this  training  is  to  hone  the 
communication  skills  between  the  forward  observer 
and  the  Fire  Direction  Center  (FDC).  In  the  current 
training  plan,  a  forward  observer  visually  locates 
targets,  identifies  and  determines  grid  coordinates 
using  binoculars  and  a  map,  and  recommends  a  call  for 
fire  to  the  FDC.  Once  the  shots  are  fired,  the  training 
instructor  (not  a  part  of  the  operational  fire  support 
team)  determines  the  accuracy  of  the  shots  and  the 
effect  on  the  target:  catastrophic  hit,  mobility  hit,  or  no 
effect.  The  calls  for  fire  are  adjusted  until  the  team  has 
the  desired  effect  on  the  target.  Before  introducing  the 
AR  system,  the  team  fired  upon  static  and  unrealistic 
proxy  targets  made  of  discarded  boxes,  tubes,  and  toy 
tanks. 

RELATED  WORK 

Our  application  of  AR  to  LVC  training  is  not  the  first, 
and  others  who  have  developed  AR  training  systems 
should  be  acknowledged.  One  early  effort  (Barrilleaux 
1999),  sponsored  by  US  Army  STRICOM  in  1993, 
combined  live  tanks  with  manned  simulators  and 
computer-generated  forces.  The  system  was 
demonstrated  in  Fort  Knox,  KY.  The  tanks  equipped 
with  limited  AR  displays  to  display  the  virtual  and 
constructive  forces  in  the  world  and  with 
instrumentation  to  send  telemetric  data  back  to  allow 
representation  in  the  virtual  and  constructive 
simulators. 

More  recently,  US  Army  STRICOM  created  a  program 
called  Embedded  Training  for  Dismounted  Soldiers 
(ETDS)  (Dumanoir  et.  al.  2002).  One  of  the  focus 
areas  of  this  program  was  to  use  wearable  computers  to 
provide  AR-  and  VR-based  training  in  the  field, 
yielding  the  MARCETE  system  (Kirkley  et.  al.  2002) 
which  integrates  an  AR  system  with  SCORM  datasets, 
and  VICTER  (Barham  et  al.  2002),  which  was  built  to 
fit  within  the  limitations  of  the  current  Land  Warrior 
system  (Natick  Soldier  Center  2001),  replacing  pieces 
of  that  system  as  necessary. 

Our  own  previous  work,  partially  funded  through  the 
ETDS  program,  includes  a  system  for  Military 
Operations  in  Urban  Terrain  (MOUT)  training, 
allowing  a  dismounted  trainee  to  navigate  a  building 
and  see  and  engage  virtual  and  constructive  enemy 
forces  in  the  real  world  (Brown  2004).  That  goal  was 
very  ambitious  and  yielded  a  proof-of-concept  system 


that  was  several  years  from  being  fielded.  The  primary 
roadblocks  are  tracking  accuracy  for  mobile 
applications  and  field  readiness  of  wearable  computers 
that  are  powerful  enough  to  drive  AR.  With  the  FiST 
training  application,  we  considered  a  problem  for 
which  a  fieldable  system  can  be  built  with  existing 
components,  that  would  benefit  from  the  real-time 
combination  of  live,  virtual,  and  constructive  forces, 
and  that  no  one  else  has  yet  addressed. 

APPLICATION  OF  AUGMENTED  REALITY 
FOR  FIRE  SUPPORT  TEAM  TRAINING 

One  of  the  goals  of  this  undertaking  was  that  the  AR 
system  should  support  the  current  training  paradigm. 
The  purpose  of  the  first  stage  is  to  hone 
communication  skills  and  not  train  for  absolute 
accuracy  in  call-for-fire.  Therefore,  the  instructor  has 
the  final  authority  over  the  success  or  failure  of  any 
particular  mortar  firing.  For  example,  the  instructor 
may  have  a  trainee  repeat  a  fire,  even  if  it  was  a  direct 
hit,  to  reiterate  the  communications  skills  learned. 

Integration  of  AR  into  the  Training  Plan 

The  AR  system,  based  on  the  Battlefield  Augmented 
Reality  System  (Livingston  et.  al.  2002),  consists  of 
two  stations  networked  together:  a  head-mounted 
display  for  the  forward  observer  and  a  touch  screen 
display  for  the  instructor.  Each  shows  the  same  set  of 
virtual  targets  superimposed  on  the  real  range.  The 
observer  station  simulates  a  view  through  a  pair  of 
binoculars  and  can  provide  a  magnified  view 
(including  a  reticle)  to  determine  target  identity  and 
grid  coordinates.  The  instructor  station  uses  a  camera 
in  a  fixed  location  to  provide  an  overall  view  of  the 
range.  The  instructor  can  start  and  stop  the  movement 
of  targets  and  determine  the  effect  of  a  fire  through  a 
simple  menu  system  and  directly  selecting  objects  on 
the  display. 

The  order  of  events  is  illustrated  in  Figure  2.  First,  the 
forward  observer,  wearing  the  AR  HMD,  observes  and 
identifies  a  target,  and  determines  its  grid  coordinates. 
Figure  3  shows  a  typical  view  of  the  virtual  targets 
overlaid  on  the  real  world.  Next,  the  observer  calls  for 
fire,  reporting  the  target  to  the  Fire  Direction  Center 
(FDC);  in  the  training,  the  instructor  also  plays  the  role 
of  the  FDC.  The  FDC  sends  an  order  to  the  mortar 
operator,  who  fires  a  real  (pneumatic)  round  at  the 
training  area.  The  instructor  looks  at  where  the  round 
landed  in  the  real  field  and  on  the  augmented  display, 
as  shown  in  Figure  4.  The  rounds  are  hard  to  see  after 
landing  on  the  field,  so  an  assistant  marks  the  round 
with  a  pole.  The  instructor  makes  a  judgment  call 
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Forward  Observer 

uses  a  tracked  see  through  head-mourned 
display  connected  to  portable  computer  The 
display  shews  virtual  targets  end  other  infor¬ 
mation  that  appear  to  exist  on  the  real 
practice  Held. 

1  identifies  and  determines  grid  coordi¬ 
nates  of  targets. 

O  calls  lor  tiro  on  target  through  the  Fire 
Direction  Corner 

6  sees  off  eel  on  target.  II  the  effect  is  not  the 
desired  D  ue.  the  FD  starts  over  at  step  1. 


uses  a  touch  screen  providing  an  augmented  view  pf  the 
Era  ining  tiei  d.  A  computer  overlays  graphics  on  the  image 
from  a  video  camera.  The  instructor  can  designate  target 
effects  simply  by  painting  to  the  targets. 

4  uses  augmented  view  of  held  to  determine  effects  of 
real  rounds  on  virtual  targets. 


Em  de  si  g  nates  effects  on  targets  so  That  f  o  rwa  rd  observer 
**  can  see  the  m  in  the  head-mounted  di  splay. 


Mortar  Operator 

operates  pneumatic  mortar  as  in 
previous  (raining  exercises. 

uses  coordinates  provided  by 
forward  observer  to  fire  a  mortar 
round. 


Figure  2.  Order  of  events  using  the  AR  system  in  training 


Figure  3.  An  augmented  view  of  the  training  area. 


Figure  5.  The  real  round  was  determined  to  have 
destroyed  the  virtual  target. 


Figure  4.  An  assistant  marks  where  round  landed. 


•  1 1 1  *  I  •  I  >  I  •  I  •  I  •  I  •  I  ^  I  •  hl'M'  I'M'  I  ■ 


r«t:i  ■  to  *:«u 
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Figure  6.  A  zooming  feature  allows  one  to  identify 
the  target. 
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about  the  effect  on  the  target  and  can  designate  that  on 
the  touchscreen  display — Figure  5  shows  a  direct  hit 
on  the  target.  Finally,  the  observer  immediately  sees 
the  designated  effect  on  the  target  in  the  HMD,  and  can 
even  zoom  in  on  the  virtual  targets  for  a  closer  look,  as 
shown  in  Figure  6  (the  video  background  is  replaced 
by  a  solid  green  background  due  to  technical 
limitations  with  the  hardware  in  use  at  the  time;  future 
versions  of  this  system  will  scale  the  video  as  well). 

System  Component  Description 

The  observer  wears  a  helmet-mounted  HMD,  as  seen 
in  Figure  7,  to  provide  a  tracked,  augmented  view  of 
the  training  area.  This  HMD  is  connected  to  a  laptop 
computer  that  drives  the  visuals.  The  HMD  contains 
cameras  just  ahead  of  the  user’s  eyes  that  collect  video 
to  be  augmented.  On  the  rear  of  the  HMD  (not  visible 
in  Figure  7)  sits  a  rear-facing  camera  used  for  high- 
precision  video-based  tracking — this  camera  captures 
images  of  a  set  of  graphical  markers  placed  behind  the 
user  and  calculates  the  position  and  orientation  of  the 
user’s  head.  With  the  high-precision  tracking,  the  user 
can  look  all  around  the  training  area  and  the  virtual 
targets  appear  to  remain  fixed  in  the  real  world.  The 
user  interface  consists  of  just  three  operations: 
controlling  the  zoom  level,  turning  the  reticle  on  and 
off,  and  turning  a  virtual  grid  on  and  off.  The 
observer’s  portion  of  the  system  is  simple  to  operate 
and  allows  the  trainee  to  concentrate  on  the  task  and 
not  the  equipment. 


Figure  7.  The  trainee  wears  a  head-mounted 


display  to  see  the  virtual  targets. 

The  instructor  uses  a  station  with  a  large,  bright  touch 
screen  attached  to  a  laptop  computer.  The  instructor 
can  start  and  stop  the  virtual  targets,  designate  effects 
on  the  targets,  and  reset  the  simulation,  through  a  few 
options  on  the  touch  screen  display.  Again,  the  focus 
was  on  simplicity:  when  the  instructor  wants  to 
designate  an  effect  on  a  target,  he  selects  the  effect 


from  a  menu  and  touches  the  target  directly.  This 
station  also  has  a  fixed  camera  with  a  wide  field  of 
view  that  collects  video  from  the  training  area  and 
sends  it  to  the  computer  to  perform  the  augmented 
overlay.  Figure  8  shows  the  instructor’s  station  as  used 
in  the  demonstration  (as  well  as  the  laptop  used  to 
drive  the  observer's  display  and  some  extra  equipment 
used  for  post-demonstration  testing  and  evaluation). 
Figure  9  shows  the  instructor  designating  an  effect  on  a 
virtual  target  using  the  augmented  touchscreen  display. 

For  this  demonstration,  all  equipment  was  loaded  onto 
a  handcart  and  powered  by  large  batteries.  We  chose 
this  path  to  keep  the  demonstration  running  all  day  and 
to  accommodate  a  lot  of  attendees  wanting  to  try  the 
system — it’s  a  lot  easier  to  put  on  a  helmet  than  an 
entire  wearable  backpack.  The  observer’s  training 
system  can  easily  run  on  the  wearable  backpack  shown 
earlier  in  Figure  1,  while  the  instructor  station  can  be 
“compacted”  onto  a  single  tablet  PC  with  an  attached 
camera. 


Figure  9.  The  instructor  designates  an  effect  on  a 
virtual  target. 
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Software  Description 

As  mentioned  previously,  the  software  was  based  on 
the  Battlefield  Augmented  Reality  System  developed 
in  our  lab.  The  BARS  libraries  were  used  to  provide 
the  augmented  reality  core  components  including 
tracker  drivers,  display  calibration,  and  video  overlay. 
Through  a  dynamic  shared  database  in  BARS  (Brown 
et.  al.  2004),  the  virtual  targets,  controlled  by  the 
instructor’s  computer,  are  also  updated  in  real  time  on 
the  observer’s  computer.  Similarly,  it  is  this 
mechanism  by  which  the  observer  sees  the  effect  on 
the  target  determined  by  the  instructor. 

For  this  particular  application,  we  added  a  few  new 
features  to  enhance  the  training  experience: 

•  Virtual  grid:  The  observer  can  turn  a  virtual 
grid  on  and  off.  This  grid  is  drawn  on  the 
ground  plane  and  is  spaced  at  simulated  1km 
intervals  (25m  actual). 

•  Terrain:  The  observer  can  turn  virtual  towns, 
roads,  and  other  artifacts  on  and  off. 

•  Zoom:  As  mentioned  previously,  the  observer 
can  zoom  in  on  the  virtual  targets. 

•  Reticle:  Also  as  mentioned  previously,  the 
observer  can  turn  a  virtual  reticle  on  and  off  to 
more  accurately  determine  the  location  of  a 
target.  If  the  reticle  is  turned  on  while 
zooming,  it  is  automatically  scaled  to  fit  the 
screen  and  allow  the  observer  to  accurately 
calculate  angles. 

One  of  the  primary  features  of  training  in  the  real 
world  using  augmented  reality  is  the  ability  to  model 
the  real  world  training  area  and  properly  occlude 
virtual  entities  as  they  move  through  the  environment. 
In  the  case  of  this  particular  training  area,  because  it 
was  a  flat  field,  there  were  no  significant  terrain 
features  to  model.  To  demonstrate  occlusion 


Figure  10.  A  real-world  object  (blue  box) 


occludes  a  virtual  tank  and  truck. 


capabilities,  we  placed  a  shipping  crate  in  the  field  and 
added  it  to  the  AR  occlusion  model.  Figure  10  shows  a 
virtual  tank  (drawn  at  the  same  1:40  scale  explained 
previously)  properly  occluded  by  the  real-world  box. 
One  can  easily  imagine  this  box  is  a  real  building,  and 
this  concept  can  be  extended  to  full-scale  hilly  or 
mountainous  training  areas  by  creating  an  occlusion 
model  from  DTED  or  similar  data. 

CONCLUSION 

Augmented  Reality  was  inserted  into  the  training  plan 
with  no  significant  changes  to  the  duties  and  actions  of 
the  participants,  except  that  they  can  now  fire  on 
moving  targets.  The  virtual  targets  for  training  were 
well  received  by  the  mortar  trainees  and  instructors  at 
Quantico.  One  USMC  captain  said: 

The  Marine  Corps  will  always  rely  on  live  training 
as  the  cornerstone  for  preparing  ourselves,  but 
simulation  and  this  type  of  augmented  reality  will 
help  make  training  more  effective  and  more 
realistic  to  live  combat.  As  we  look  to  develop 
requirement  documents  for  range  instrumentation 
and  improved  MOUT  facilities,  AR  will  be  a 
technology  that  we  incorporate  into  the 
appropriate  aspect  of  the  training  facilities. 

However,  rigorous  studies  and  measurements  of 
effectiveness  are  yet  to  be  done.  The  system  can  also 
insert  virtual  terrain  and  control  measures  into  the 
display,  and  both  capabilities  were  preliminarily  tested 
at  Quantico.  Future  plans  include  refining  the  system, 
using  multiple  and/or  pan-tilt-zoom  cameras, 
implementing  the  system  at  a  full-scale  live  fire  range 
such  as  Twenty  nine  Palms,  and  completing  a  Semi- 
Automated  Forces  (SAF)  interface  for  more  intelligent 
targets. 
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